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I. REAL PARTY IN INTEREST 

The real party in interest for this appeal and the present application is Xerox 
Corporation, by way of an Assignment recorded in the U.S. Patent and Trademark Office 
beginning at Reel 012275, Frame 0344. 
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II. STATEMENT OF RELATED APPEALS AND INTERFERENCES 

There are no prior or pending appeals, interferences or judicial proceedings, known to 
Appellants, Appellants' representative, or the Assignee, that may be related to, or which will 
directly affect or be directly affected by or have a bearing upon the Board's decision in the 
pending appeal. 
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III. STATUS OF CLAIMS 
Claims 1-20 are on appeal 
Claims 1-20 are pending. 
No claims are allowed. 
Claims 1-20 are rejected. 
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IV. STATUS OF AMENDMENTS 

A Request for Reconsideration was filed on March 8, 2007 in reply to the Final 
Rejection mailed January 24, 2007. The Request for Reconsideration was considered by the 
Examiner, as evidenced by the Advisory Action mailed March 28, 2007. 
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V. SUMMARY OF CLAIMED SUBJECT MATTER 

The claims are directed to apparatus and methods for generating a summary of a 
captured (e.g., scanned) image of a document on the basis of detected handwritten or 
electronic annotations made to the machine generated text of the document prior to scanning 
(see Figs. 1-4; page 2, lines 26-29). The captured image is processed to detect annotations 
made to the original text of the document prior to image capture (see Figs. 2-4; page 3, line 
29 to page 4, line 17). The detected annotations can be used to identify features or text for 
use to summarize that document (see Fig. 4; page 4, lines 18-31; page 5, line 21). The 
apparatus "for generating a summary of a document" of independent claim 1 includes "an 
image capture device for capturing an original image of a document" (see Figs. 3, 4). The 
apparatus also includes "a processing device for detecting annotations made to the document 
prior to image capture" (see Fig. 4; page 5, lines 4-16). The annotations identify "lesser 
portions of text to be extracted from machine generated text of the original image of the 
document" (see Figs. 3, 4; page 5, lines 4-16). Importantly, the processing device determines 
"the lesser portions of text of the document identified by the annotations to be extracted, and 
extracts] only the lesser portions of text from the machine generated text from the original 
image of the document identified by the detected annotations" (see Figs. 3, 4; page 5, line 17 
to page 6, line 12). The apparatus further includes "a summary generator for generating a 
summary including only the extracted lesser portions of text from the machine generated text 
of the document" (see Fig. 4; page 5, line 21 to page 6, line 12). The summary is "a 
condensed version of the machine generated text of the original image of the document and 
[is] generated as a separate electronic image document that is different from the original 
image" (page 6, lines 11 and 12). 

The method "of generating a summary of a document" of independent claim 8 
includes "capturing an original image of a document" (see Figs. 3, 4), and "detecting 
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annotations made to the document prior to image capture" (see Fig. 4; page 5, lines 4-16). 
The annotations identify "lesser portions of text to be extracted from machine generated text 
of the original image of the document" (see Figs. 3, 4; page 5, lines 4-16). Importantly, the 
method also includes "determining the lesser portions of text of the document identified by 
the annotations to be extracted," and "extracting only the lesser portions of text from the 
machine generated text from the original image of the document identified in the determining 
step" (see Figs. 3, 4; page 5, line 17 to page 6, line 12). The method further includes 
"generating a summary including only the lesser extracted portions of text from the machine 
generated text of the document" (see Fig. 4; page 5, line 21 to page 6, line 12). The summary 
is "a condensed version of the machine generated text of the original image of the document 
and [is] generated as a separate electronic image document that is different from the original 
image" (page 6, lines 11 and 12). 

Figs. 1 and 2 illustrate a general schematic block diagram and process for processing a 
paper document to generate a summary of the document encompassed by claims 1-6 and 8- 
13. Fig. 3 generally illustrates a page of a document with annotations made to the machine 
generated text of the original image encompassed by claims 1-6 and 8-13. Fig. 4 is an 
enlarged schematic view of a portion of Fig. 3 illustrating the extraction of a sentence that has 
been annotated encompassed by claims 1-6 and 8-13. Fig. 5 illustrates various options for 
displaying the summary encompassed by claims 7, 14-16, 19 and 20. 

In particular, for example, referring to Fig. 3, various kinds of handwritten annotations 
are made to the machine generated text of the original image 12. These include underlining 
32, circling 34, bracketing 36, margin bracketing or marking 38, cross-through 40, anchored 
arrows indicating place changes 42, and handwritten notes or insertions 44 (page 4, lines 14- 
17). Importantly, the text or other features indicated by the annotations are extracted and used 
to compile the summary of areas of interest identified by the hand annotations (page 4, lines 
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3-6; page 5, lines 4-7). For example, as shown in Fig. 4, the extracted feature for an 
annotation is preferably expanded to include additional contextual information or text for that 
annotation. That is, the feature may be expanded to include the sentence 50 around the 
annotation (page 5, lines 10-16). The summary is compiled from the extracted features, for 
example, in the form of image maps of the extracted features (page 5, lines 21-24). The 
summary therefore reflects the areas of interest identified by the hand annotations in the paper 
document. 

As shown, for example, in Fig. 2, the process encompassed by the claims includes a first 
step 30 of identifying the annotations made by the user in the captured digital image (page 4, 
lines 7-9). At step 46, interpretation of the annotations is carried out (page 4, lines 18-20). 
At step 48, the machine generated text in regions of the digital image identified by the 
interpreted annotations are extracted (page 5, lines 4-9). At step 52, the summary is compiled 
from the extracted features (page 5, lines 21-24). At step 56, the compiled summary is 
output, for example, on a user's display or it is printed (page 6, lines 1 1 and 12). 

As illustrated in Fig. 5, for example, the apparatus and methods can provide a plurality 
of layered detail levels in a window 57 for the summary. For example, the lowest detail level 
58 merely includes any subject headings extracted from the document. By clicking on any 
subject heading, the subject heading is expanded to a second detail level 60 to generate the 
text summary of that appropriate section of the document. The second detail level 60 only 
includes text features. However, by clicking again, the summary is expanded (a third detail 
level 62) to include non-text features as part of the summary, such as annotated figures from 
that section of the document. By clicking on any sentence, the summary is again expanded (a 
fourth detail level 64) to display further context for the sentence, for example, by displaying 
the paragraph containing the sentence. In a final layer (a fifth detail level 66), the annotation 
associated with any sentence in the document may be "retrieved" by clicking on the sentence. 
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VI. GROUNDS OF REJECTION TO BE REVIEWED ON APPEAL 

The following grounds of rejection are presented for review: 

(1) Claims 1-6, 8-13, 17 and 18 are rejected under 35 U.S.C. §103(a) over Manly, 
U.S. Patent No. 3,714,636, in view of Ma et al. (Ma), U.S. Patent No. 6,909,805. 

(2) Claims 7, 15 and 16 are rejected under 35 U.S.C. § 103(a) over Manly in view of 
Bornstein et al. (Bornstein), U.S. Patent No. 5,867,164. 

(3) Claim 14 is rejected under 35 U.S.C. § 103(a) over Manly in view of Kupiec, U.S. 
Patent No. 6,533,822. 

(4) Claims 19 and 20 are rejected under 35 U.S.C. §103(a) over Manly in view of 
Lerner et al. (Lerner), U.S. Patent No. 6,859,909. 
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VII. ARGUMENT 

The Final Rejection fails to establish a prima facie case of obviousness with respect to 
the current rejection of pending claims 1-6, 8-13, 17 and 18. In particular, the Final Rejection 
fails to (a) apply prior art references that teach or suggest all of the claim limitations of at 
least independent claims 1 and 8; and (b) provide suggestion or motivation, either in the 
references themselves or in the knowledge generally available to the skilled artisan, to modify 
the reference or to combine reference teachings. Further, the rejections of claims 7, 14-16, 19 
and 20 are legally defective because, due to the fact that these claims depend from claim 1 or 
claim 8, the Patent Office cannot reject these claims without using at least the references that 
were used to reject claims 1 and 8. 

A. Requirements for a Prima Facie Case of Obviousness under 35 U.S.C* 
§103(a) 

In order to establish a prima facie case of obviousness, three criteria must be met: (1) 
there must be some suggestion or motivation, either in the references themselves or in the 
knowledge generally available to the skilled artisan, to modify the reference or to combine 
reference teachings; (2) there must be a reasonable expectation of success; and (3) the prior 
art must teach or suggest all of the claim limitations {In re Vaeck, 947 F.2d 488 (Fed. Cir. 
1991); see MPEP §2142, §2143). The criteria must be supported in the prior art, and not 
based on Appellants' disclosure. 

B. The Final Rejection fails to apply prior art references that teach or 
suggest all of the claim limitations of at least the independent claims 

Appellants respectfully submit that the Final Rejection fails to satisfy at least the third 

criteria for a prima facie case of obviousness. Specifically, and as required by MPEP 

§2143.03, the Final Rejection has failed to apply prior art references that teach or suggest all 

of the claim limitations of at least independent claims 1 and 8 (see In re Royka, 490 F.2d 981 

(CCPA 1974)). Moreover, the Office Action fails to allege that any features missing from the 
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references would have been generally known in the art. MPEP §2143.03 states that all words 
in a claim must be considered in judging the patentability of the claim against the prior art 
(see In re Wilson, 424 F.2d 1382 (CCPA 1970)). Here, although the Final Rejection alleges 
that the combination of Manly and Ma suggests all of the claim features recited in 
independent claims 1 and 8, the Final Rejection does not establish that all claimed features 
are present in the references. 

First, with respect to Manly, Appellants respectfully submit that, contrary to the Final 
Rejection's assertion, Manly fails to disclose or suggest extracting only the lesser portions of 
text from the machine generated text identified by detected annotations, and generating a 
summary including only the extracted lesser portions of text from the machine generated text 
of the document, as recited in independent claims 1 and 8. Manly discloses a manually 
operated typewriter that generates machine readable text on an original document (col. 1, 
lines 13-22). An author that proofreads the original document makes editing marks directly 
on the original document with a special pen (col. 1, lines 23-28). The editing marks of the 
special pen designate text of the original document to be deleted or removed (col. 1, lines 32- 
41), and instruct a machine to delete or remove the designated text (col. 1, lines 28-41). That 
is, the editing marks of the special pen instruct the machine to extract unwanted or 
undesirable portions of text from the original document. The deleted or removed (i.e., 
unwanted or undesirable) text is discarded and not included in the author's final draft. The 
extracted, deleted text is not included in any subsequently generated summary, as is the case 
in claims 1 and 8. Moreover, to include the extracted Manly test in a summary would be 
contrary to the teachings of Manly. 

Claims 1 and 8 recite extracting portions of text identified by detected annotations to 
be included in a summary, while Manly teaches extracting portions of text identified by 
detected annotations to be discarded. The Examiner acknowledges in the Advisory Action 
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that Manly teaches extracting only unwanted or undesirable portions of text from the original 
document, and that the extracted text is discarded so as not to be included in the authors final 
draft. However, the Examiner asserts in the Advisory Action that by indicating with the 
special pen portions of text to be extracted and discarded, the author would necessarily be at 
the same time designating text of the draft that would be included in the final draft. The 
Examiner further asserts that the final draft corresponds to the claimed generated summary. 

However, as clearly recited in claims 1 and 8, the claimed generated summary only 
includes those portions of the text from the machine generated text that are extracted because 
they are identified by detected annotations . Accordingly, any portions of the text in Manly 
not designated by the special pen are irrelevant to the affirmatively claimed features of this 
application. The not-designated portions of Manly are not extracted. The not-designated 
portions constitute the final document, which is the antithesis of a summary of a document. 
The combinations of features recited in claims 1 and 8 allow a user to affirmatively annotate 
important or relevant (e.g., desired) sections of text of an original document image to be 
subsequently compiled in a summary of the original document. Manly does not disclose these 
features or even the same concepts or goals obtained by the claimed features. 

Manly discloses edit signals that instruct a machine to insert material that is separately 
generated by the original typewriter (col. 1, lines 47-50). However, the insert material 
("insert record") is additional material that is "separately generated" (col. 1, lines 54-61) by 
the original typewriter to be inserted in the final record. Thus, the inserted, additional 
material is not designated by the editing marks of the special pen, and does not designate 
lesser portions of the text to be extracted . Moreover, the "insert record" is included in the 
final document, and is not a summary of the document. Thus, Manly ? s insert material also 
does not correspond to the claimed extracted lesser portions. 
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Thus, Manly fails to disclose or suggest extracting only lesser portions of text from 
the machine generated text identified by detected annotations, and generating a summary 
including only the extracted lesser portions of text from the machine generated text of the 
document, as recited in independent claims 1 and 8. 

Second, and as acknowledged in the Final Rejection, Manly fails to disclose 
generating a summary that is a condensed version of the machine generated text of the 
original image of the document, the summary being generated as a separate electronic image 
document that is different from the original image, as recited in independent claims 1 and 8. 
The Office Action alleges that Ma discloses these features. 

Ma teaches that after extraction from the document image, all the printed text lines 52 
may be saved into a memory (col. 7, lines 52-61). Ma does not disclose or suggest that the 
memory only includes the extracted lesser portions of text from the machine generated text of 
the document, or that the information stored in the memory is a condensed version of the 
machine generated text of the original image of the document. Instead, the entire original 
image of the document is saved into a memory (col. 7, lines 52-61). Accordingly, Ma is 
unable to allow only important or relevant sections of the original document image designated 
by handwritten annotations to be compiled as a summary of the original image of the 
document, which is a benefit of the combinations of features recited in claims 1 and 8. 

Further, Ma teaches that the printed text lines 52 may be saved into a memory (col. 7, 
lines 52-61). Ma does not teach that the printed text lines 52 are subsequently output to form 
a separate electronic image document . Moreover, because all the printed text lines 52 (the 
entire original image of the document) are saved into a memory, as discussed above, Ma also 
fails to teach that any separate electronic image document would be different from the 
original image. Therefore, Ma fails to disclose or suggest a summary generated as a separate 
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electronic image document that is different from the original image, as recited in claims 1 and 
8. Further, Ma fails to disclose generating a summary that is a condensed version of the 
machine generated text of the original image of the document, the summary being generated 
as a separate electronic image document that is different from the original image, as recited in 
independent claims 1 and 8. Thus, the combination of Manly and Ma fails to suggests all of 
the claim features recited in independent claims 1 and 8. Accordingly, the rejection is 
improper. Appellants respectfully request withdrawal of the rejection. 

According to MPEP §2143.03, if an independent claim is non-obvious under 35 
U.S.C. § 103(a), then any claim depending therefrom is non-obvious (In re Fine, 837 F.2d 
1071, (Fed. Cir. 1988)). Thus, withdrawal of the rejections of the remaining, dependent 
claims also is respectfully requested. 

C. The Final Rejection fails to provide suggestion or motivation, either in the 
references themselves or in the knowledge generally available to the 
skilled artisan, to modify the reference or to combine reference teachings 

Appellants respectfully submit that the Final Rejection fails to satisfy the first criteria 

for a prima facie case of obviousness. Specifically, and as required by, MPEP §2143.01(V) 

and MPEP §2143.01(VI), the Final Rejection has failed to provide suggestion or motivation, 

either in the references themselves or in the knowledge generally available to the skilled 

artisan, to modify the reference or to combine reference teachings. MPEP §2143.01(V) states 

that if the proposed modification of a reference would render the invention being modified 

unsatisfactory for its intended purpose, there is no suggestion or motivation to make the 

proposed modification (In re Gordon, 733 F.2d 900 (Fed. Cir. 1984)). MPEP §2143.01(VI) 

states that if the proposed modification of combination of the prior art would change the 

principle of operation of the prior art invention being modified, then the teachings of the 

references are not sufficient to render the claims prima facie obvious (In re Ratti, 270 F.2d 

810 (CCPA 1959)). 
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The Final Rejection asserts that it would have been obvious to one skilled in the art to 
modify the system and method of Manly by the alleged teachings of Ma to result in the 
combination of features recited in independent claims 1 and 8. However, Appellants 
respectfully submit that such a modification of the method of Manly would render the system 
and method of Manly unsuitable for their intended purpose as well as change Manly's 
principle of operation. 

As discussed above, Manly teaches proofreading an original document by making 
editing marks that designate unwanted or undesirable text of the original document to be 
deleted or removed. The deleted text is discarded and not included in the authors final draft 
while the undeleted text is the author's final draft. On the other hand, as discussed above, Ma 
teaches eliminating all the printed text lines from the scanned document image, irrespective 
of the location of handwritten annotations (see Fig. 1, steps 24 and 26; col. 3, line 67 to col. 4, 
line 5; col. 5, lines 64-66; col. 7, lines 53-61; Figs. 7 and 8). The Final Rejection's proposed 
modification of a Manly would render the system and method of Manly unsatisfactory for its 
intended purpose because Manly's proofreading process seeks to extract only unwanted or 
undesirable text and maintain the desired text as a final draft, but Ma eliminates all the 
printed text lines. Further, the proposed modification of Manly with the teachings of Ma 
changes the principle of operation of the Manly's system and method because extracting all 
the printed text from the scanned document image defeats the purpose of proofreading a 
document to eliminate only undesirable text of a draft. 

Both the Final Rejection and the Advisory Action allege that the motivation to 
combine would be to increase "the flexibility of having multiple formats, and efficient 
separation of document data" (see Ma, col. 2, lines 30-37). However, the Final Rejection has 
not cited any evidence or support in either Ma or Manly that explains, discusses or suggests 
how the process of eliminating all the printed text lines from the scanned document image, 
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irrespective of the location of handwritten annotations, as taught by Ma, would make the 
proofreading process of Manly, in which only the undesirable text from the original document 
is deleted or removed, more efficient. Moreover, the additional "format" disclosed by Ma is 
not applicable to the proofreading process of Manly because, as discussed above, extracting 
all the printed text from the scanned document image defeats the purpose of proofreading. 

Although the Final Rejection concocts an alleged benefit of incorporating the process 
of Ma into the proofreading procedure of Manly, there is no evidence or support that explains, 
discusses or suggests how the two methods would be combined without defeating the 
disclosed purpose of Manly. Thus, the Examiner has provided no viable evidence to support 
the conclusion that one skilled in the art would have been motivated to combine Manly and 
Ma. Instead, the Office Action relies on impermissible hindsight using knowledge gleaned 
only from Appellants' disclosure (see MPEP §2145(X)(A)). 

Because, the modification of Manly proposed by the Final Rejection would render the 
system and method of Manly unsuitable for its intended purpose and/or the Final Rejection's 
proposed modification of the system and method of Manly would change the principle of 
operation of Manly, there can be no motivation for such a modification (MPEP §§ 
2143.01(V) and 2143.01(VI)). Because there is no motivation to modify Manly, the Final 
Rejection has failed to establish a prima facie case of obviousness under 35 U.S.C. § 103(a). 
Accordingly, the rejection is improper. Appellants respectfully request withdrawal of the 
rejection. 

D. The rejections of claims 7, 14-16. 19 and 20 are legally defective 

The rejections of claims 7, 14-16, 19 and 20 are legally defective because the Final 
Rejection rejects these claims without using all the references that were used to reject 
independent claims 1 and 8. Specifically, none of the rejections of claims 7, 14-16, 19 and 20 
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rely on Ma (combined with Manly). Thus, the rejections of claims 7, 14-16, 19 and 20 are 
legally defective. Accordingly, withdrawal of the rejections is respectfully requested. 
E. Conclusion 

Because the Final Rejection fails to apply prior art references that teach or suggest all 
of the claim limitations of at least independent claims 1 and 8, and because there is no 
motivation to modify Manly as proposed by the Final Rejection, the Final Rejection has failed 
to establish a prima facie case of obviousness under 35 U.S.C. § 103(a). Accordingly, the 
rejections of claims 1-20 are improper. 
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VIII. CONCLUSION 

For at least the reasons discussed above, it is respectfully submitted that the rejections 
are in error and that claims 1-20 are in condition for allowance. For at least the above reasons, 
Appellants respectfully request this Honorable Board to reverse the rejections of claims 1-20. 
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APPENDIX A - CLAIMS APPENDIX 

CLAIMS INVOLVED IN THE APPEAL: 

1. An apparatus for generating a summary of a document, comprising: 

an image capture device for capturing an original image of a document; 

a processing device for detecting annotations made to the document prior to 
image capture, the annotations identifying lesser portions of text to be extracted from 
machine generated text of the original image of the document, the processing device 
determining the lesser portions of text of the document identified by the annotations to be 
extracted, and extracting only the lesser portions of text from the machine generated text from 
the original image of the document identified by the detected annotations; and 

a summary generator for generating a summary including only the extracted 
lesser portions of text from the machine generated text of the document, the summary being a 
condensed version of the machine generated text of the original image of the document and 
being generated as a separate electronic image document that is different from the original 
image. 

2. The apparatus according to claim 1, wherein the summary generator is 
operative to generate a summary of the same document as that on which the annotations are 
detected. 

3. The apparatus according to claim 1, wherein the summary generator is 
operative to generate a summary of a different document as that on which the annotations are 
detected. 

4. The apparatus according to claim 3, wherein the image capture device is 
operative to capture an image of a second document to be summarized based on the detected 
annotations from a first document. 



A-1 



Application No. 09/982,024 

5. The apparatus according to claim 1, wherein the processing device is operative 
to identify an image region associated with a detected annotation. 

6. The apparatus according to claim 5, wherein the image region represents a 
sentence in the document image to provide context for the identified annotation. 

7. The apparatus according to claim 1, wherein the summary generator is 
operative to generate a summary comprising hidden portions, each portion selectively 
expandable to increase the information in that portion of the summary. 

8. A method of generating a summary of a document, comprising: 
capturing an original image of a document; 

detecting annotations made to the document prior to image capture, the 
annotations identifying lesser portions of text to be extracted from machine generated text of 
the original image of the document; 

determining the lesser portions of text of the document identified by the 
annotations to be extracted; 

extracting only the lesser portions of text from the machine generated text 
from the original image of the document identified in the determining step; and 

generating a summary including only the lesser extracted portions of text from 
the machine generated text of the document, the summary being a condensed version of the 
machine generated text of the original image of the document and being generated as a 
separate electronic image document that is different from the original image. 

9. A method according to claim 8, wherein the document summarized is same 
document as that on which the annotations are detected. 

10. A method according to claim 8, wherein the document summarized is a 
different document from that on which the annotations are detected. 
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11. A method according to claim 10, further comprising capturing an image of a 
second document to be summarized based on the detected annotations from the first 
document. 

12. A method according to claim 8, wherein said detection comprises identifying 
an image region associated with a detected annotation. 

13. A method according to claim 12, wherein the image region represents a 
sentence in the document image to provide context for the identified annotation. 

14. A method according to claim 8, further comprising generating a summary 
comprising hidden portions, each portion selectively expandable to increase the information 
in that portion of the summary. 

15. The apparatus according to claim 1, wherein the summary includes expandable 
detail levels. 

16. The method according to claim 8, wherein generating the summary includes 
providing the summary in expandable detail levels. 

17. The apparatus according to claim 1, wherein the summary includes extracted 
image maps of the original image of the document. 

18. The method according to claim 8, wherein generating the summary includes 
extracting image maps of the original image of the document. 

19. The apparatus according to claim 1, wherein the summary includes at least one 
of a pointer and address indicating a place in the original image of the document from which 
the extracted portions are extracted. 

20. The method according to claim 8, wherein generating the summary includes 
providing at least one of a pointer and address indicating a place in the original image of the 
document from which the extracted portions are extracted. 
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